Optimising Hardware Accelerated Neural Networks with Quantisation and a Knowledge Distillation Evolutionary Algorithm

This paper compares the latency, accuracy, training time and hardware costs of neural networks compressed with our new multi-objective evolutionary algorithm called NEMOKD, quantisation. We evaluate NEMOKD on Intel’s Movidius Myriad X VPU processor, quantisation Xilinx’s programmable Z7020 FPGA hardware. Evolving models increases inference accuracy by up to 82% at cost 38% increased throughput performance 100–590 image frames-per-second (FPS). Quantisation identifies a sweet spot 3 bit precision in trade-off between requirements, accuracy. Parallelising implementations 2 quantised from 6 k FPS 373 FPS, 62× speedup.